fix(graphical): EP statistics completion — F6 truncated KL + F7(b) evidence recording#1354
Merged
Merged
Conversation
…idence recording (#1353) Closes out the #1332 audit's statistical findings: - F6: TruncatedNormalMessage.kl now computes the exact same-support KL via truncated moments (scipy truncnorm.stats); reduces to the untruncated Gaussian formula as bounds recede. The old untruncated approximation distorted the EPHistory convergence metric as posterior mass approached the bounds (live in HowToFit ch3 via af.TruncatedGaussianPrior). MC-verified at interior, near-bounds and half-bounded geometries. - F7(b): the search-driven factor projection now records the sampler's log-evidence of the tilted fit on the projected mean field's log_norm (AbstractSearch.optimise -> MeanField.from_priors(log_norm=...)), implementing the design README §5 already documented. Both levels guarded (StaticResult has no samples; MCMC/MLE have no evidence -> 0.0 default). With F7(a) (#1351) and F7(c) (#1345) this makes EPMeanField.log_evidence trustworthy for model comparison when factor searches are nested samplers. - README §5: stale audit warning replaced with the F7-fixed statement + the nested-sampler requirement; §6: #1336 Recommendation-A decision recorded (keep all three deterministic-composition mechanisms, trade-off documented, compound pattern is the declarative surface). Adds test_ep_statistics_completion.py (7 tests: MC KL parity x3 geometries, wide-bounds reduction, near-bounds divergence from the old formula, support mismatch raise, from_priors log_norm plumbing). Full suite: 1474 passed / 14 skipped / 1 pre-existing order-dependent failure (test_full_hierachical, triaged on #1352 — deterministic test-order dependence, not a flake; unrelated to this diff). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes out the #1332 audit's statistical findings (tracker #1353). F6:
TruncatedNormalMessage.klnow computes the exact same-support KL from truncated moments — the old untruncated approximation distorted theEPHistoryconvergence metric as posterior mass approached the bounds (live in HowToFit ch3 viaaf.TruncatedGaussianPrior); the new expression reduces to the Gaussian formula as bounds recede and is Monte-Carlo-verified at interior, near-bounds and half-bounded geometries. F7(b): the search-driven factor projection now records the sampler's log-evidence on the projected mean field'slog_norm(AbstractSearch.optimise→MeanField.from_priors(log_norm=…)) — implementing the design README §5 already documented. With F7(a) (#1351) and F7(c) (#1345), all three legs of "evidence bookkeeping broken" are now fixed:EPMeanField.log_evidenceis trustworthy for model comparison when factor searches are nested samplers (MCMC/MLE factors contribute 0 — documented).Also carries the #1336 Recommendation-A decision into README §6 (keep all three deterministic-composition mechanisms; tighter-vs-modular trade-off documented; compound pattern is the declarative surface) and replaces §5's stale audit warning.
API Changes
Behaviour fixes + one optional argument.
TruncatedNormalMessage.klreturns the exact truncated KL (convergence-metric values change for truncated-prior EP fits — correctly).MeanField.from_priorsgainslog_norm=0.0.EPMeanField.log_evidencebecomes meaningful in sampler-driven fits (previously silently missing all sampler evidence).See full details below.
Test Plan
test_full_hierachical, now triaged on test_full_hierachical flaky — hierarchical EP lands at wrong mu_logt #1352 as deterministic test-order dependence (0/6 standalone with bit-identical value, 3/3 whole-file; id-counter → EP-update-order hypothesis posted); unrelated to this difftest_ep_statistics_completion.py— 7 tests: MC-vs-analytic truncated KL at 3 geometries, wide-bounds reduction to the Gaussian formula, near-bounds divergence from the old formula, support-mismatch raise,from_priorslog_norm plumbing (incl. through the fix(graphical): EP statistics fix batch — F1/F2/F4/F8 from #1332 #1351 operator algebra)ep_parityinitially failed on a stale resume cache — degenerate reloaded samples, joint sigma=0; fresh run passes; noted for the cache-hygiene file)test_staticregression caught in-suite and fixed (two-level getattr:StaticResulthas nosamples)Validation checklist (--auto run — plan on the issue)
Full API Changes (for automation & release notes)
Added
MeanField.from_priors(priors, log_norm=0.0)— optional evidence carried by the projected mean field (the per-factorẐₐof README §5 Eq. 14)Changed Behaviour
TruncatedNormalMessage.kl(dist)— exact same-support truncated KL via truncated moments (scipy.truncnorm.stats); previously the untruncated Gaussian formula. Different-support case still raises. EP convergence-metric values on truncated-prior graphs change (correctly)AbstractSearch.optimise(EP factor updates) — the projected mean field'slog_normis the sampler's log-evidence of the tilted fit (0.0 for searches without an evidence estimate, and for results with no samples, e.g.StaticResult)autofit/graphical/README.md— §5 audit warning replaced (F7 fixed; nested-sampler requirement stated); §6 records the research: EP review phase 5 — reconcile the three deterministic-variable mechanisms #1336 Recommendation-A decisionMigration
EPMeanField.log_evidencevalues across versions should note they were previously missing all sampler evidence.🤖 Generated with Claude Code